Patterns manufactured by semiconductor processes in recent years have advanced in miniaturization and improvement of pattern overlay accuracy over multiple layers of an exposure apparatus is required. It is considered that the importance of measuring the overlay with high accuracy and feeding it back to the exposure apparatus will be more increased. PTL 1 discloses an electron microscope that measures an overlay error between patterns formed in different layers by overlaying a reference image and an image to be measured. Particularly, PTL 1 discloses a method of extracting a pattern for each layer according to a difference in luminance for each the reference image and the image to be measured, performing an overlay between the reference image and the image to be measured using a pattern matching method, and calculating a positional deviation between patterns of both the overlaid images. In addition, PTL 2 discloses a method of measuring a pattern located at a plurality of heights by detecting backscattered electrons or the like obtained when a sample is irradiated with a high-acceleration electron beam of 50 kV or higher. Specifically, a method of measuring a dimension between a pattern formed on the sample and a buried pattern using electrons emitted from a plurality of patterns located at different depths (heights) is disclosed.